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ABSTRACT 


In FY 1987, fuzzy set mathematical techniques were applied to the problem of remote 
viewing (RV) analysis. Two analytical methods were developed: the first was designed to be 
sensitive to the verbal content of the RV response; the second was designed to account for the 
visual/spatial arrangements of response elements. A definition of “ ground truth,” against which 


these new analytical techniques could be tested, was also devised. 


The verbal method is predicated on the application of fuzzy set mathematics to the figure 
of merit (FM) technology.* It also features a new descriptor list, which was introduced to 
provide a richer vocabulary for analysis. The list’s hierarchical structuring in levels, ranging from 
very abstract to very concrete, affords considerable flexibility for analytical manipulation of 
descriptor elements. A pilot application of the verbal analysis was shown to correlate highly with 
ground truth. 


The combination of fuzzy set technology and the new descriptor list also proved effective 
for the visual/spatial approach. The implementation of these techniques, in conjunction with a. 
third technique known as “cluster analysis,” has resulted in an algorithm for the production of 
orthogonal target sets. This has resulted in a significantly more effective rank—order analysis 
procedure. 


The visual and verbal analyses were each determined to have certain strengths and 
weaknesses. The verbal analysis can be statistically more powerful with good RV responses and 
provides a more comprehensive breakdown of the verbal information in an RV response. It is 
quite labor intensive to apply, however, and it appears to be relatively insensitive to noisy RV 
data. “Noisy,” in this context, defined as a preponderance of incorrectly identified response 
elements. The visual analysis system is inherently much less powerful statistically and is less 
capable of providing systematic objectification of the true RV signal content. It can be rapidly 
applied however, and is sensitive to the primary manifestation of true RV signal in noisy 
data--namely, to the visual arrangement of RV response elements, regardless of their verbal 
labels. Potential applications of these techniques in their current states have been suggested; 


areas of future research for their refinement have been identified. 


, The FM analysis has continued to undergo refinement since its inception in FY 1984. 
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I INTRODUCTION 


Since the publication of the initial remote viewing (RV) effort at SRI International,*! two 


basic questions have remained in evaluating remote viewing data: 


® What is the definition of the target? 


® What is the definition of the RV response? 


_ In the development of meaningful evaluation procedures, we must address these two 
questions. 


In the older, IEEE-style, outbound experiment, definitions of target and response were 
particularly difficult to achieve. The protocol for such an experiment dictated that an 
experimenter travel to some randomly chosen location at a prearranged time; a viewer’s task was 
to describe that location. In trying to assess the quality of the RV descriptions (in a series of 
trials, for example), an analyst visited each of the sites and attempted to match responses to 
them. While standing at a site, the analyst had to determine not only the bounds of the site, but 
also the site details that were to be included in the analysis. To cite a specific example using this 
protocol: if the analyst were to stand in the middle of the Golden Gate Bridge, he/she would 
have to determine whether the buildings of downtown San Francisco, which are clearly and. 
prominently visible, were to be considered part of the Golden Gate Bridge target. The RV 
response to the Golden Gate Bridge target could be equally troublesome, because responses of 
this sort were typically 15 pages of dream-like free associations. A reasonable description of the 
bridge might be contained in the response; it might be obfuscated, however, by a large amount of 


unrelated material. How was an analyst to approach this problem of response definition? 


The first attempt at quantitatively defining an RV response involved reducing the raw 
transcript to a series of declarative statements called concepts.2 Initially, it was determined that a 
coherent concept should not be reduced to its component parts. For example, a small red VW 
car would be considered a single concept rather than four separate concepts, small, red, VW, 
and car. Once a transcript had been “conceptualized,” the list of concepts constituted, by 
definition, the RV response. The analyst rated the concept lists against the sites. Although this 


represented a major advance over previous methods, no attempt was made to define the target 


* 
References are listed at the end of this réport. 
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site. It was also extremely labor intensive and did not readily allow for rapid processing of RV 
data. 


During an FY 1982 program, a procedure was developed to define both the target and 
response material.? It became evident that before a site can be quantified, the overall remote 
viewing goal must be clearly defined. If the goal is simply to demonstrate the existence of the RV 
phenomena, then anything that is perceived at the site is important. But if the goal is to gain 
specific information about the RV process, then possibly specific items at the site are important 
while others remain insignificant. For example, let us assume that an office is a hypothetical 
target and that a single computer in that office is of specific interest. Let us also assume that a 
viewer gives an accurate description of the shape of the office, provides the serial number of the 
typewriter, and gives a complete description of the owner of the office. Although this kind of a 
response might provide excellent evidence for remote viewing, the target of interest (the 
computer) is completely missed. This response, therefore, does not yield information about the 


role of complex target elements at the site. 


In FY 1984, work began on a computerized evaluation procedure,4 which underwent 
significant expansion and refinement during FY 1986.5 The mathematical formalism underlying 
this procedure is known as the “figure of merit” (FM) analysis. This method is predicated on 
descriptor list technology, which represented a significant improvement over earlier “conceptual 
analysis” techniques, both in terms of “objectifying” the analysis of free response data and in 


increasing the speed and efficiency with which evaluation can be accomplished. 


It became increasingly evident, however, that this particular application of descriptor lists 
was inadequate in providing discriminators that were “fine” enough to describe a complex target 
accurately; it was also unable to exploit fully the more subtle or abstract information content of 
the RV response. To decrease the granularity of the RV evaluation system, therefore, it was 
determined that the technology would have to evolve in the direction of allowing the analyst a 
gradation of judgment about target and response features, rather than the hard-edged, (and 
rather imprecise) all-or-nothing binary determinations. A preliminary survey of various 
disciplines and their evaluation methods (spanning such diverse fields as artificial intelligence, 
linguistics, and environmental psychology) revealed a branch of mathematics, known as “fuzzy 


set theory,” which provides a mathematical framework for modeling situations that are inherently . 


imprecise. 
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This report describes the FY 1987 analysis program,* which focused on the application of 
fuzzy set mathematics to descriptor list technology. In addition, an effort was made to further 
reduce granularity by redesigning the descriptor list itself, and to introduce an altogether new 
adjunct technology for capturing visual/spatial RV information. A separate but parallel research 


line investigated the application of expert systems technology to the problem of RV analysis. t 


" This report constitutes Objective A, Task 3, “Fuzzy Set Applications in Remote Viewing 

Analysis 

T The results of the expert systems approach | to RV analysis can be found in the FY 1987 
Final Technical Report. 
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II METHOD OF APPROACH 


The method of approach was aimed at accounting for both the verbal and visual aspects of 
RV data. In this section we present the techniques that were developed to address these 


objectives. 


A. Target Pool Preparation 


A few preliminary guidelines governing target pool composition were distilled from two 
sources: (1) the opinions of RV monitors about the appropriateness of various kinds of RV 
targets for novice viewers, and (2) RV analysts’ assessments about the difficulties encountered in 


using descriptor lists to score National Geographic Magazine target materials. 


As a general rule, the current subjective consensus is that targets are inappropriate for 


experimental purposes if they exhibit any of the following qualities: 


e They are contrary to the viewer’s expectations. (For example, if the target 
pool is basic gestalt scenes, then there will not be a living room scene.) 


@ They are imbued with negative emotional impact.* 
e They violate the “spirit” of the descriptor list’s intended use. 


® They exhibit descriptors that are ambiguously depicted. 


Every effort was made in FY 1987 to create a target pool of 200 National Geographic 
Magazine photographs that did not exhibit any of the deficiencies enumerated above. Particular 
attention was paid to ensuring that all descriptors were unequivocally depicted: a given target 
would be rejected a priori, for example, if its particular photographic angle exhibited what was, 
in fact, a Jake (a completely-bounded body of water) as a bay (a partially—bounded body of 
water). This approach was taken in order to minimize the potential for disputes concerning the 
visual content of any given photograph: in this way, it was anticipated that the feedback 
experience for the viewer would be unencumbered, as much as possible, by visual versus 


conceptual ambiguities. 


* Laboratory anecdotal evidence suggests that targets having negative emotional impact often 
result in psi-missing responses. 
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B. Fuzzy Set Theory 


Fuzzy set theory was chosen as the focal point of the RV analytical techniques, because it 
provides a mathematical framework for modeling situations that are inherently imprecise. 
Because it is such an important component of both the verbal and visual analyses, a brief tutorial 


from an earlier report5 will be recapitulated to highlight its major concepts. 


1. A Tutorial 


In traditional set theory, an element is either a member of a set or it isn’t--e.g., the 
number 2 is a member of the set of even numbers; the number 3 is not. Fuzzy set theory is a 
variant of traditional set theory, in that it introduces the concept of degree of membership: herein 
lies the essence of its applicability to the modeling of imprecise systems. For example, if we take 
the concept of age (known as a linguistic variable in fuzzy set parlance), we might ascribe to it 
certain subcategories (i.e., fuzzy sets) such as very young, young, middle-aged, old, etc. Looking 
at very young only as a fuzzy set example, we must define what we mean by this concept vis—a-vis 
the linguistic variable age.* If we examine the chronological ages from 1 to 30, we might 
subjectively assert that we consider the ages 1 through 4 to represent rather well a spectrum of 
the concept very young, whereas the age of 30 probably does not accurately represent very young 
at all. As depicted in Figure 1, fuzzy set theory allows us to assign a numerical value between 0 
and 1 that represents our best subjective estimate as to how much each of the ages 1 though 30 
embodies the concept very young. 


Clearly, a different set of numerical values would be assigned to the ages 1 through 30 
for the fuzzy sets young, middle-aged, and old——e.g., the age of 6 might receive a value of 0.5 
for very young, but a value of 1.0 for young, depending on context, consensus, and the particular 
application of the system. In this way we are able to provide manipulatable numerical values for 
imprecise natural language expressions; in addition, we are no longer forced into making 


inaccurate binary decisions such as, “Is the age of 7 very young—-yes or no?” 


* It is important to note that the design of the fuzzy application occurs in accordance 
with the subjectivity of the system designer. Fortunately, the fuzzy set technology is 
rich enough that it allows for a virtually unrestricted range of expression. Technically 
speaking, young is the fuzzy set and very is the modifier, but it is beyond the scope of 
this paper to present terminology in depth. 
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Young > Ciliddie-agedy Co 


Very young 


0.01 0.0 
02000 eevee 30 


FIGURE 1 THE FUZZY SET “VERY YOUNG” 


2. Summary of Definitions 


In this example, the set of ages ranging from 1 through 30 is a crisp set and can be 
defined as the universe of elements. By crisp, we mean, for example, that the age “30 years and 
1 day” would fall outside the universe of elements by definition. Very young, young, 
middle-aged, and old are fuzzy sets drawn from this universe of elements. They are called fuzzy 
because they contain members from the universe of elements that exhibit varying degrees of 
membership, ranging from partial to complete. These terms are important for understanding the 
design of the current descriptor list and the mathematical formalism underlying its 


implementation. 


C. Descriptor List Design 


The primary impetus in redesigning the descriptor list was to reduce its granularity by 
affording a richer vocabulary of terms for response and target interpretation. The following 
sections outline (1) the derivation of the list’s universe of elements, and (2) the design 


philosophy that resulted in their hierarchical arrangement. 


1. The Universe of Elements 


The universe of elements—-i.e., the entire list of descriptors shown in Figure 
2--represents the union of the set of elements in the National Geographic Magazine target pool 
of 200 and the set of elements obtained from the FY 1987 RVs for which this pool supplied the 
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targets. This set of elements was obtained in a blind fashion--i.e., in the absence of knowledge 
as to which responses correctly matched which targets. In the case of the RV transcript—derived 
elements, an effort was made to preserve the vocabulary used by the viewers. Some of the 
descriptors, therefore, are either response—dependent or target-dependent or both, while others 


(particularly at the more abstract levels) appear to be more potentially universal across descriptor 
lists. 


2. Descriptor List Structure 


The universe of elements is structured in /evels, ranging from the relatively abstract 
(information poor) to the relatively complex (information rich). The current system has the 
capability of using seven primary and three secondary levels of descriptors: the main intent of this 
structure is to serve as a heuristic device for guiding the analyst into making judicious concrete 
descriptor assignments based on rather abstract commentary. A secondary goal is to make an 
attempt at objectifying the analyst’s unconscious decision-making process. The use of levels is 
advantageous in that each descriptor level can be weighted separately and used or not, as the 
case may be, in a computerized data base. This enables various combinations of levels to be 
deployed, in order to identify the optimal mix of concrete versus abstract descriptors. Optimal 
here might be defined as the highest figures of merit, or perhaps by some other measure, such as 


a consensus rating of the experimental series in question. 


The determination as to which descriptors belonged on which level was made after 
consideration of two primary factors: (1) the apparent ability of the viewers to be able to resolve 
certain features, coupled with (2) the amount of pure information thought to be contained in any 
given descriptor. In the first case, for example, it appears that viewers have greater difficulty on 
average in perceiving waterfalls as opposed to rivers. Some of these “factor one” determinations 
were based on the combined observational subjective lore, obtained by both analysts and 
monitors in the course of either analyzing or conducting numerous RV experiments; some were 
determined empirically from post hoc analyses of viewers’ abilities to perceive various descriptor 


elements in previous experiments. 
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The “factor two” determinations were made primarily by arranging the descriptors 
such that a descriptor at any given level represents the sum of constituent descriptors at lower 
levels. The world is not a very crisp place and not all of its elements are amenable to hierarchical 
structuring: certain violations of the “factor two” rule appear, therefore, throughout the 
proposed levels. It should be noted, however, that some of the more glaring violations were 
largely driven by the “factor one” determinations (i.e., the viewers’ abilities to discern certain 
elements) enumerated above. 


3. Target and Response Definitions 


The descriptor list shown in Figure 2 was filled out for each of the 200 targets, 
according to a consensus (see Section III) reached by three analysts on each of the descriptors. 
This approach was used to mitigate the potential influence of any single coder’s biases and 
idiosyncrasies. A numerical assignment (1, OS p< 1) was made for each descriptor in 
response to the following question: 


How visually important is this descriptor to this target? 


Each target’s coded list served as its formal definition for the purposes of both verbal 
and visual analysis. It should be noted that the descriptor set was devised to define targets in 
terms of visual importance. If the analysis were to be re-framed along a different dimension of 
interest--e.g., conceptual, functional, allegorical, etc.--the descriptor set in Figure 2 would have 
to be revised to address the new mission. * 


For responses in which verbal analysis was used, pu assignments for each descriptor 
were made in response to a different question: 


To what degree am I convinced that this descriptor represents this response 
element? 


Responses were coded according to this different criterion (but still using the list in 
Figure 2), because RV response composites are often not detailed enough to determine relative 
visual importance for any given element. For the purposes of verbal analysis,t the coded 


descriptor lists represented the formal definition of each response. 


a ee 
* Details of these analyses can be found in Sections D. and E., below. 


t An altogether different response definition was used for the purposes of visual analysis. 
This is discussed in Section E. 
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The assigned q's, themselves, were one-digit fuzzy numbers between 0 and 1 (e.g., 
.1, .2, .3, etc.). In some rare cases, two-digit assignments (e.g., .15, .25, .35, etc.) were made; 


any finer assignments, however, were deemed to be meaningless. 


D, Verbal Analysis: the Fuzzy Set Figure of Merit 


The current RV verbal analysis is predicated on the application of fuzzy set mathematics to 
the figure of merit technology. 


1. Fuzzy Set Definition of Figure of Merit 


Once the fuzzy sets that define the target and the response have been specified, the 
comparison between them to provide an FM is rather straightforward. In previous work,§ we 
have defined Accuracy as the percent of the target material that was described correctly by a 
response. Likewise, we have defined Reliability (of the viewer) as the percent of the response 
that was correct. The FM is the product of the two; to obtain a high FM, a viewer has to 
describe a large portion of the target material correctly in as parsimonious a way as possible. 


These quantities for the jth target/response pair are as follows: 


2 W, (RJNT)), 


Accuracy; = a; = : 
p> Wy Ti, 
k 
DW, (RAT) 
Reliability, = r, = — ; 
> Wy Rix 
k 
and 
Figure of Merit; = Mj = aj xX T;, 


where the sum over k is called the sigma count in fuzzy set terminology, and is defined as the 
sum of the membership values, HM, for the elements of the response, target, or their 
intersection—~intersecting Rj, Tj, and (RiNT}), respectively. A fuzzy intersection is defined as the 
minimum of the two membership values. In this version of the figure of merit definition, we have 
allowed for the possibility of weighting the membership values, Wx, in order to examine various 


linguistic contributions to the FM. 
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For the above calculation to be meaningful, the 1’s for the targets must be similar in 
kind to the yw’s for the responses. As we noted above, this is not the case. The target ’s 
represent the visual importance of the element relative to the scene, and the response p’s 
represent the degree to which the analyst is convinced that the element is represented in the 


response. 


With advanced viewers it might be possible to change the definition of the response 
’s to match the definition of the target ’s. In this case, the viewer must not only recognize that 
an element is present in the target, but must also provide information as to how visually important 
it is. This ability is beyond the skill of most novice viewers. Alternatively, we have opted to 
modify the target y definition, by employing the fuzzy set technique of a-cuts. An a-cut is a 
way to set a threshold for visual importance, so that all target elements possessing that threshold 
value or higher are considered to be full members of the target set. In fuzzy set parlance, an 
a@-cut converts a fuzzy set to a crisp set. The result is that the target set is now devoid of visual 
information: a potential target element is either present or absent in the target set, regardless of 
its actual visual importance. Even with this conceptual change in the target definition, the FM 
formalism described above remains applicable, because a crisp set can be considered as a fuzzy 
set with all membership values equal to 1. It should be noted that the @-cut conversion does not 
occur in the data base proper, but rather in the analysis itself. In this way the integrity of the 
original visual importance information can be maintained, and the a@~cut can be exercised as a 
variable. 


2. P-Values for a Given Figure of Merit 


It is difficult to arrive at a general assessment as to how well a given response matches 
a specified target. The ideal situation is to obtain some absolute measure of goodness of match. 
While the FM is an approximation to this measure, it provides no information as to the likelihood 
of a particular FM value. The answer to that question is nearly impossible, because it requires 
knowledge of the viewer’s specific response bias for the session. It is possible to determine 
general response biases,+ but that knowledge is only useful on the average. For example, a 
viewer may love rock climbing and may spend most of her/his free time involved in that activity, 
Thus, the general response bias would probably entail aspects of mountains, rocks, ropes, etc. 
Suppose, however, that the viewer Spent the evening previous to a given RV session on a 
romantic moonlight sail on San Francisco Bay. For this specific RV session, the response bias 


might be apt to include romantic images of the moonlit water, lights of the city, and bridges. 


The current solution to the problem is to provide a relative assessment of FM 


likelihood (i.e., a differential measure). A relative assessment addresses the following question: 
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“How good is the response matched against its intended target, when compared to all possible 
targets that could have been chosen for the session?” Unfortunately, the answer depends upon 
the nature of the remaining targets in the pool. An example of the worst-case scenario illustrates 
the problem. Suppose that the target pool consisted of 100 photographs of waterfalls, and the 
viewer gave a near-perfect description of a waterfall. Suppose further that this viewer had been 
well calibrated so that we can assume his/her description was not fortuitous. An absolute 


assessment of the resulting FM should be good, whereas a relative assessment will be low. 


The worst-case scenario can be avoided, to a large degree, by carefully selecting the 
target pool. In section E.1 we describe an application of fuzzy set theory that addresses target 


similarities vis-a-vis the problem of target pool optimization. 


To provide a relative assessment of the likelihood of a given FM, we define the 
p-value for the session as the fraction of targets out of 200 that have an FM equal to or higher 
than the given FM. 


Consecutive RV responses by the same viewer are not altogether statistically 
independent. The statistically independent random element in the session is the target. 
Therefore using the p-value defined above maintains the trial-by-trial independence. The mean 


chance expectation for the p-values is a unit uniform distribution with a mean of 0.5. 


We are currently exploring various methods for combining the session p—values to 


provide an overall assessment of a series of viewings. 


E. Visual Analysis: An Application of Cluster Technology 


The verbal analysis appears to afford a reasonably comprehensive estimate of RV response 
information along a certain dimension: namely, identifying what the response elements are, 
ranging from abstract to concrete descriptions. This analysis is quite insensitive, however, in 
accounting for the spatial arrangement of the response elements. It has long been the subjective 
opinion of analysts that it is the visual arrangement of response elements~—irrespective of what 
these elements are actually called by the novice viewer--that comprise the major part of the true 
RV signal line. This hypothesis was the major impetus for the development of an analysis system 


that would account for this visual/spatial type of RV information. 


1. Target Similarities 


Target definition for the purposes of this mode of analysis is exactly the same as the 


one that was used for the verbal analysis—~i.e., a given target is defined by its descriptor list, 
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which has been coded to reflect the visual importance of each target element. Since the average 
number of elements that have been assigned a non-zero value is approximately 37, the fuzzy set 
representation of the target pool is rich in visual information. We used this information to 


determine the degree to which the target set contains visually similar targets. 


It is beyond the scope of this report to describe the extensive work in the literature 
seeking to find algorithmic techniques that mimic human assessments of visual similarity. One 


recent article describes techniques similar to the one we used.® 


We begin by defining the similarity between target j and target k (Sj,x) to be a 
normalized fuzzy set intersection between the two target sets: 


(= W; (Tat): ) 


py W, Ti, > Wi Ty, 
1 


j,k 


For N targets there are N(N-1)/2 unique values (19,900 for N=200) of Sik. The value j and k 
that correspond to the largest value of Sj,x represent the two targets that “look” most similar. 
Suppose another target (m) is chosen and Sm,j and Sm,k are computed. If both of these values 
are larger than Sm,n (for all n not equal to j or k) then target m is assessed to be most similar to 


the pair j,k. The process of grouping targets based on these similarities is called cluster analysis. 


Table 1 provides an overview of the 19 clusters found from the total analysis of the 
200 targets. Figure 3 shows the graphic output of a single cluster in detail: a much more 
complex--and visually difficult--graph is generated for the full cluster analysis (see Appendix 
B); this smaller subset, therefore, has been chosen to be illustrative of the whole analysis.* All 
targets in this particular sample cluster are islands; the island in each photograph is visible in its 
entirety. Except for one outlier (i.e., a hexagonal building covering an island), the islands fall 
into two main groups, i.e., with and without manmade elements. The natural islands include 


three similar mountain islands, two sandbars, and two flat verdant islands. 


* In order e make the graphic output more meaningful, we actually did the analysis 
with 1 - Six . 
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Table 1 
NAMES OF THE 19 CLUSTERS 


Flat Towns Cities with Prominent Geometries 
Waterfalls Snowy Mountains 

Mountain Towns Valleys with Rivers 

Cities with Prominent Structure Meandering Rivers 

Cities on Water Alpine Scenes 

Desert/Water Interfaces Outposts in Snowy Mountains 
Deserts Islands 

Dry Ruins Verdant Ruins 

Towns on Water Agricultural Scenes 

Outposts on Water 


COwomrnrnn bw ND 


os 


Target 1xxx | 


: , Number 
Linear Geometries 


(e.g., Runways) 


Many Structures 
(e.g., Town) 


Flat And Verdant 


Sand Bars 


1161 
Mountains 1049 


1038 


Hexagonal Building Covering Island——————> 1003 
Da a a 
0.0 0.2 0.4 0.6 0.8 
ae 


FIGURE 3. DETAILED CLUSTER ANALYSIS OF THE ISLAND CLUSTER 
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2. RV Analysis Using Orthogonal Targets 


The visual analysis of RV data has been the primary method used in simple 
rank-order judging. In the past, the technique involved a ranking of the actual targets in a series 


against a given response. There are two possible problems with this approach: 


(1) The targets are not optimized for judging discrimination. 


(2) Long series are difficult to assess. 


Using the results of the cluster analysis, it is possible to select a number of decoy 
targets that are all different from each other and from the intended target, as well. The 
technique we have chosen involves a random selection of 6 out of 18 possible decoy clusters; the 
intended target cluster is not used. One target is randomly selected from each of these six decoy 
clusters. The intended target is added to the group of six decoys, and they are all numerically 
ordered on target number; these are then presented with the single response to the analyst for 
assessment, 


The analyst’s task is to rank order the targets from the best to the worst match based 
on a subjective assessment of the visual correspondence. Analysts are essentially instructed to 
pay primary attention to the drawings in a response (particularly, in terms of perspective) and 
secondary attention to the attendant verbal commentary, i.e., the written material is not to be 
ignored completely, but its contribution is small. The rationale for this is evident in the overall 
intent of the visual/spatial analysis. Analysts are also instructed to regard incongruent or 
surprising response elements as more salient, and to be slightly biased in favor of elements 
occurring at the beginning of a response. The reasons for these instructions are less evident and 
are largely attributable to the observations and subjective impressions of experienced and 
successful analysts. It was also determined that a consensus approach is useful for objectifying 
the decision-making process: for the FY 1987 data, all rankings were determined by a 
consensus of two analysts. For a series of many sessions, the sum of ranks for the actual matches 


is computed and a standard sum-of-ranks distribution is used to compute the p-value for the 


series. 


The primary advantages to this approach are the following: (1) an analyst never has 
to provide an assessment for more than seven items at a time, and (2) the analysis can be rapidly 
applied for very long experimental series. The most important advantage, however, is that the six 
decoys and the intended target look as differently as possible, given the nature of the target pool 
at large and the fact that decoys must be selected randomly. The application of fuzzy set 
technology has provided an algorithm for the production of orthogonal target sets: this is 


perceived as the primary contribution to the success of this analysis technique. 
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F. Development of Ground Truth 


To determine whether any of the new analytical approaches were effective, a standard had 
to be developed against which they could be measured. It was determined that this 
standard--known as “ground truth”--should consist of a “real world” normalized consensus as 


to the degree of correspondence between RV responses and their intended targets. 


To achieve this objective, each person from a sample population of 37 was presented, on 
an individual basis, with six remote viewing responses and their associated targets. This set 
comprised an experimental series for Viewer 177 in an FY 1986 photomultiplier tube (PMT) 
experiment.’ Each response was matched to its intended target; this created six such sets, 
numbered sequentially from 9001 through 9006. The responses were fairly typical of novice 
viewer output: they consisted of two to five pages of rudimentary drawings with some associated 
descriptive words. The targets consisted of six photographs of outdoor scenes selected from an 


FY 1986 National Geographic Magazine target pool. 


Each person was asked individually for his/her subjective judgment as to the degree of 
correspondence between the remote viewing responses and their respective targets. This was to 
be accomplished without the benefit of a given definition of “degree of correspondence;” 
respondents were free to formulate their own criteria. The only information provided was that 
responses typically begin (on the first page or pages) with small bits of information and eventually 


culminate in a composite drawing at the end. 


After this brief introduction to the task, each individual was instructed to examine all of 
the responses and their intended targets. Then, on a session—by—session basis, the subject was 
asked: (1) to assess the degree of correspondence between the remote viewing response and its 
intended target, and (2) to register this correspondence assessment by making a vertical hash 
mark across a 10 cm scale ranging from “none” to “complete.” Figure 4 provides a sample of 


the form that the respondents used to indicate their six assessments. 


After completing the assessment task, respondents were then asked to furnish some minor 
autobiographical information (i.e., age, occupation, male/female, handedness, belief in ESP). 
Degree of belief in ESP was also registered on a 10 cm scale: each individual simply made a 
vertical hash mark at the appropriate point on a_ continuous line between “no belief” and 


“complete belief.” 
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ANALYSTS’ INSTRUCTIONS FOR REMOTE VIEWING SERIES 900X 


Thank you for helping us perform a post hoc assessment of a series of remote viewings. 
The targets were actually 35 mm slides that were attached to a photomultiplier, a device to 
measure small amounts of light. We were searching for possible physical correlates to remote 
viewing. 

You will find in your packet six remote viewing responses labeled 9001-9006, respectively. 
Also shown is the target number of the intended photograph. We have supplied the original, 
rather than the 35 mm slide. 


We would like you to make a subjective judgment as to the degree of correspondence 
between the remote viewing response and its associated target. Familiarize yourself with the task 
by first looking at all the responses and their intended targets. Then, on a session—by—session 
basis, rate your assessments. You are completely free to define what is mean by “Degree of 
Correspondence.” Indicate your judgment by marking one line across the appropriate 
continuous scale shown below. A vertical line near the “None” end of the scale will indicate that 
you feel there is very little correspondence between that response-target pair. Likewise a vertical 
line near the “Complete” end of the scale will indicate that you feel that there is a significant 
degree of correspondence. 


Many of the responses begin with a little information and build toward a composite drawing 
at the end. Please assess the response in its entirety as best you can. Thank you again. 


SESSION DEGREE OF CORRESPONDENCE TARGET 


Complete 


te | 1034 
bh t042 


1065 
1094 
1005 
1024 

Age: a 

Occupation: 

Male/Female: 


Handedness (R/L) 


Belief in ESP: -—___—_—_____| 


None Complete 


FIGURE 4 SAMPLE GROUND TRUTH FORM 
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To perform the ground truth analysis, distance measurements were taken from the left end 
point of each scale to the vertical slash mark for each assessment. These measurements were 
then entered into a computerized data base. The first step in analyzing this aggregate of data was 
to compute the mean and standard deviation for each individual’s set of assessments. In order to 
nullify response bias, each analyst’s set was normalized by a z-score transformation. For a given 
target/response pair, all analysts’ data could then be combined. The mean and standard 
deviation were then computed over all analysts’ assessments for each target/response pair. 
Figure 5 shows these normalized means for each target/response pair: these constitute the basis 
for the ground truth against which other techniques were measured. See Section III, Results. 
The solid bars illustrate the degree of positive correspondence between the remote viewing 
responses and their intended targets, while the shaded bars. represent the degree of negative 
correspondence (i.e., worse than average for this set of six). Note that these are relative to the 


others in the set. All six correspondences in the set could be good. 


0.5 
Normalized ~ ~ ae 


ean 
-0.5 


9001 9002 9003 | 9004 9005 9006 


Session Number 


FIGURE 5 NORMALIZED MEAN FOR EACH TARGET/RESPONSE PAIR 
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III RESULTS 


To effect a meaningful comparison between ground truth and the verbal analysis, the same 
PMT series that served as the ground truth set was also analyzed by the fuzzy set figure of merit 
method. The verbal fuzzy sets (\1’s) for the six targets and six responses were consensus—coded 
by five analysts ranging from expert to novice. A typical spread of p assignments was -£0.1 with 
an occasional outlier. Some of the elements were vigorously debated until a consensus was 
reached, Figures of merit were calculated for each target/response pair, and all possible unique 
differences (n(n—1)/2 = 15; n = 6) were computed. These differences were then correlated 
against similar differences computed from ground truth (described above). 


FM (a-cut=0.2) 
Ground Truth 


r= 0.720 


band 
re) 
“A 
aay 
s 
i=) 
a 
Lod 
° 
I 
= 
a 
3 
<t 
o 


6 


Session Difference ID 


FIGURE 6 GROUND TRUTH COMPARED TO VERBAL FM 


Figure 6 shows the results of this calculation where RV quality difference for verbal is 


shown as open circles and ground truth quality difference as triangles. The visual qualitative 


22 


Approved For Release 2000/08/08 : CIA-RDP96-00789R002200140001-2 


Approved For Release 2000/08/08 : CIA-RDP96-00789R002200140001-2 


agreement is supported by the linear correlation coefficient between the two sets of points of 
0.720, 


This implies that the combination of (1) the structure of the universe of elements (i.e., the 
verbal hierarchical structure), (2) the fuzzy set mathematics, and (3) a consensus approach to 
assessing the fuzzy sets themselves, provided an accurate representation of the subjective scoring 
of the same data by a large number of randomly selected individuals. In short, the FY 1987 
verbal analysis system provided output that closely correlated with ground truth. 


To summarize, the visual and verbal analyses each have their own respective strengths and 
weaknesses. The verbal analysis is statistically more powerful and provides a more 
comprehensive breakdown of the verbal information in an RV response. It is quite labor intensive 
to apply, however, and it appears to be relatively insensitive to noisy RV data.* “Noisy,” in this 
context, is defined as a preponderance of incorrectly identified response elements. The visual 
analysis system is statistically much less powerful and is less capable of providing systematic 
objectification of the true RV signal content. It can be rapidly applied, however, and is sensitive 
to the primary manifestation of true RV signal in noisy data-~namely, to the visual arrangement 
of RV response elements, regardless of their verbal labels. It was the consensus of experimental 
monitors that the FY 1987 data were relatively noisy. Therefore, although the visual analysis 
system has yet to be formally compared to ground truth, it was chosen as the primary analytical 
method for the FY 1987 RV experiments, 


ee 
“ The PMT experimental series chosen for the pilot verbal analysis application turned 
out to be considerably more robust than much of the data obtained from the FY 1987 
experiments. 
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IV RECOMMENDATIONS AND CON CLUSIONS 


A. Possible Research Areas for Refinement of the Current Systems 


Several future research areas are suggested for thé refinement of the current analytical 
systems. One area that needs to be examined systematically is the use of both inter- and 
intra-level weighting factors. For the purposes of the FY 1987 analyses, all levels and descriptors 
were accorded equal weight. The ideal goal would be to determine the optimal weighted mix of 


abstract versus concrete elements, as a means to achieving the following objectives: 


(1) refinement of the cluster analysis for targets, in an effort to simulate, as closely 
as possible, what is meant by “visual similarities” between targets, and 


(2) refinement of the verbal analysis of responses, in an effort to achieve even 
greater correlations between the fuzzy set figure of merit analyses and various 
forms of ground truth. 


Another area that requires examination in some detail is the set of descriptors (i.e., the 
universe of elements), and the hierarchical nature of its structure. It is probable that some 
descriptors are more appropriate than others; furthermore, they might be more effectively 
Structured in a semantic network as opposed to a true hierarchy. If a hierarchical structure is 
retained, then some attention must be paid to the formulation of logical consistency rules that 
govern descriptor use. This would include numerical relationships governing the values (1’s) of 
higher-order descriptors (e.g., port) vis-a-vis the combined value of their constituent parts 


(e.g., city, river, boats, jetties, commercial, etc.). 


One perceived inadequacy of the verbal system is that it atomizes conceptual “units.” For 
example, if the response element is red box, information is lost in separating red from box. 
Current research in fuzzy set theory indicates that fuzzy aggregates of fuzzy elements——“ fuzzy 
sets of fuzzy sets”~~are mathematically complex but possible. Some effort should be made to 
determine whether this technology could be implemented, as a means to capturing the 
information content of the RV response with greater accuracy. 


For the visual analysis, research into visual similarities between pictures of natural scenes 
may serve as a potential refinement tool. The aim here would be to enhance the visual 


orthogonality of the decoy targets as much as possible. Experiments in normal perception of 
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similarities would assist in determining whether scenes are perceived as similar because of their 
low-level geometries, concrete elements, or some combination of factors. Should the lower, 
abstract levels prove important to similarities perceptions, then Fourier decomposition of target 
materials might assist in providing more accurate target y’s at these levels. In any case, the 
ultimate aim would be to refine the target cluster analysis such that it closely simulates ground 
truth perceptions of orthogonality. 


B. Possible Applications of the Current Systems 


The verbal and visual analytical methods exhibit different strengths and weaknesses and 
appear to account for different aspects of the RV signal line. Even without further refinement or 
modification, the systems appear to be quite sensitive and accurate in their respective domains: 


the verbal system, for example, correlated highly with collected ground truth. 


This suggests that they might be used effectively in their current state for certain 
applications. One possibility is that the two methods might be used in combination to afford 
operative definitions of “advanced” versus “novice” RV functioning. One might predict, for 
instance, that verbal scores for novices would be relatively low, while their visual scores would be 
somewhat higher. An advanced viewer, on the other hand, might be one who consistently 
produces scores in both areas above certain predetermined thresholds. It is also possible that this 
approach might eventually provide a formal definition for RV “learning.” As the two analytical 


systems are deployed and refined, other applications may be identified for systematic 
investigation in the future. 
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INSTRUCTIONS FOR ANALYSTS 
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ANALYSTS’ GUIDE TO THE FY 1987 NATIONAL GEOGRAPHIC 
DESCRIPTOR SCHEME 


A. Introduction 


This document is intended to assist analysts in the assessment of novice level remote 
viewings (RVs) of National Geographic Magazine targets. As we are all aware, novice transcript 
data can be quite sparse or abstract or both: these characteristics have often resulted in 
considerable loss of information within the framework of the previous twenty-bit descriptor list. 
In FY 1987, we have attempted to reduce the granularity of the analysis system by incorporating 
the use of fuzzy set mathematics and by introducing the ability to interpret responses and targets 
in more abstract terms. 


The degree of abstraction and atomization that transcripts and targets can usefully tolerate 
are open questions at present. One method of exploring these parameters involves the use of 
levels of descriptors, ranging from the relatively abstract (information poor) to the relatively 
complex (information rich). The proposed new evaluation system has the capability of using 
seven primary* and three secondary levels of descriptors, through which we hope to parse and to 
objectify the chain of logic that analysts employ for making decisions about concrete descriptors 
based on rather abstract commentary. The intent here, of course, is to make an attempt at 
bringing the analyst’s unconscious decision-making process out of the murky realm of 
superstition and magic and into the hard, cold, scientific light of day! The use of levels is 
advantageous in that each descriptor level can be weighted separately and used or not, as the 
case may be, in a computerized data base. This will enable various combinations of levels to be 
deployed, in order to identify the optimal mix of concrete versus abstract descriptors. Optimal 
here might be defined as the highest figures of merit or perhaps by some other measure, such as a 


consensus rating of the experimental series in question. 


It is anticipated that most of the potential problems will occur on the abstract descriptor 
end of the spectrum: how abstract can a descriptor be and still be useful? To illustrate the 


potential problems posed by increasingly abstract descriptors, consider the fact that all responses 


* Consider the elegant symmetry here between the descent into hell, Dante-style, and the 


appropriateness of having seven primary levels of analysis. 
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and targets contain some admixture of the two highly abstract elements horizontal lines and 
vertical lines; to perform fuzzy assessments along these dimensions, therefore, may not enhance 
the distinguishability of targets and responses. On the other hand, a concept such as rectangular 
forms, while still quite abstract, is only selectively present in transcripts and targets: it may prove 
useful, therefore, for capturing data outright that have heretofore been acknowledged only tacitly 
via the process of interpretation. It may be, therefore, that the ideal system would encapsulate a 
degree of greater abstraction, without adding levels of descriptors that are so abstract as to be 
universally represented in all responses and targets. In any case, this is what the new system 
proposes to explore systematically. In the next section an overview of the descriptor levels will be 


provided. 


B. An Overview of Descriptor Level Theory 


The entire list of descriptors represents the union of the set of elements in the National 
Geographic Magazine target pool of 200, and the set of elements obtained from the FY 1987 
RVs for which this pool supplied the targets. This set of elements was obtained in a blind 
fashion-—i.e., in the absence of knowledge as to which responses correctly matched which 
targets. In the case of the RV transcript-derived elements, an effort was made to preserve the 
vocabulary used by the viewers. Some of the descriptors, therefore, are either 
response—dependent or target-dependent or both, while others (particularly at the more abstract 
levels) appear to be more potentially universal across descriptor lists. The determination as to 
which descriptors belonged on which level was made after consideration of two primary factors: 
(1) the apparent ability of the viewers to be able to resolve certain features, coupled with (2) the 
amount of pure information thought to be contained in any given descriptor. In the first case, for 
example, it appears that viewers have greater difficulty on average in perceiving waterfalls as 
opposed to rivers. Some of these factor one determinations are based on the combined 
observational subjective lore, obtained by both analysts and monitors in the course of either 
analyzing or conducting numerous RV experiments; some have been determined empirically 
from post hoc analyses of viewers’ abilities to perceive various descriptor elements in previous 


experiments. 


The factor two determinations were made primarily by arranging the descriptors such that 
a descriptor at any given level represents the sum of constituent descriptors at lower levels. 
Obviously, the world is not a very crisp place, and so there appear to be violations of this rule 
throughout the proposed levels. Keep in mind, however, that certain of the more glaring 
violations are largely driven by the factor one determinations (i.e., the viewers’ abilities to 


discern certain elements) enumerated above. 
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For the purposes of our experiments, the higher the level in number, the higher the 
weighting factor (W)--which is to say that more concrete information merits more credit. 
Fortunately, the analyst need not concern himself with the Ws, because these consist of 
predetermined multipliers built into the mathematical structure of the system. The Ws consist of 
a two-dimensional weighting system that reflects mission-dependency: in this case, the overall 
mission is to accord greater weight to more specific kinds of information. The secondary mission 
is to provide a mechanism for assessing the viewer’s (or viewers’) ability to resolve a given 


descriptor within a given level. 


The numerical values that the analyst will have to concern himself with are the values 
given to the individual descriptors themselves (1’s). For targets, the ,’s will be assessed on a 
descriptor-by-descriptor basis according to the presence or absence of each descriptor in terms 
of visual importance. For responses, the w’s will be assessed according to the analyst’s best 
estimate as to whether each descriptor is present or absent at all. The concept of visual 
importance is not relevant for novice-level responses, because the data are often not detailed 


enough to form meaningful composite pictures.* 


Cc. The Descriptor Levels and Their Use 


With this basic theory in mind, it is appropriate at this juncture to provide an overview of 
what each level typifies and how the descriptors in these levels might be used by the analyst. All 
subsequent commentary, therefore, is in reference to the actual descriptor sheets themselves, 
which will be used for encoding the FY 1987 targets and responses. As a general comment on 
some of the mechanics of the sheets, descriptors in parenthesis, are simply alternate words that 
may be encountered in lieu of the main descriptor--e.g., rise (slope); descriptors in brackets, 
will probably be encountered only rarely--e.g. [oasis]. 


Perhaps the simplest way to obtain an overview of the seven descriptor levels is to proceed 


from the bottom up—-i.e., from the simplest or most abstract, to the most complex or concrete. 


1. Abstract Descriptor Levels II: Levels 1 and 2 


Levels 1 and 2 are provided to account for the abstract geometry present primarily in 
responses and secondarily in targets. Level 1 contains one-dimensional geometric elements and 


is aimed at the most primitive RV ideograms. For example, the following situation might occur: 


* More complete information on the mechanics of making the p assignments is provided in 
Section D. 
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The viewer draws Labels it And The target is 
this: this: this: 
“Stepped” | | 
and 
“City” “Mesa” 


It would seem reasonable to assume here, that a mesa might be adequately described as a 
stepped feature and that the viewer’s initial ideogram represented some amount of “true” signal. 
In the previous twenty-descriptor system, there was no mechanism for giving the viewer credit for 
the validity of his raw ideogram, despite the fact that he misinterpreted its meaning. In the new 
system, the viewer can receive Level 1 credit straightaway for the perception of stepped. The 
Level 1 weighting factor (which the analyst need not worry about) will probably be small, because 
the Level 1 elements are primitive, ubiquitous, and information poor. As a result, the viewer 
won't receive very much credit for a primitive perception like stepped, but this would appear to 
be reasonable: logic would dictate that a higher order perception such as mesa should in fact be 
worth more credit. 


Level 2 consists of combined 2-D and 3-D geometries. The rationale for combining 
the two geometries resides in the observation that viewers’ drawing skills are primitive: a box, for 


example, usually has no more representational information than a rectangle. 


The principal problem for the analyst in using levels 1 and 2 will be in attempting to 
decompose concrete target elements into abstract geometries, such that some meaningful overlap 
between targets and responses can be achieved. Fortunately, fuzzy set theory holds promise for 
ameliorating some of this difficulty, in that it will enable the analyst to make small numerical 
assignments for “low confidence” descriptors. For example, if the target were a series of 
mountain peaks, an analyst might be willing to make small numerical assignments for the highly 
stylized descriptors of inverted V-shape, repeat motif, triangle, and perhaps, cone (if the scene 
looked particularly volcanic). In the optimal situation for target coding, there would be 
descriptor lists constructed for each viewer, such that the list used for any given viewer would be 
specifically tailored to the vocabulary and ideogramic idiosyncrasies of that viewer alone. In this 
way, targets could be abstractly decomposed in a given viewer’s own terms. Unfortunately, we do 
not have the luxury of doing this at present, because one list must suffice for a group of viewers. 
The governing rules in making assignments at levels 1 and 2, therefore, would appear to be the 
following: 


(1) Targets should be as liberally but defendably interpreted (albeit with 
probably small fuzzy values) across as many geometric descriptors as 
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seem reasonable, in order to be inclusive of all the viewers’ potential 
drawings and debriefs, while 


(2) Response geometries should simply be chronicled in the descriptor list as 
they are literally observed. Level 2 descriptors such as rectangle are not 
to be parsed into the constituent parts of horizontal lines and vertical 
lines. To assign js’s for “constituent parts” bits, response elements have 
to be clearly and discretely identifiable as these things in and of 
themselves. 


In most cases, there are probably both literal and stylized descriptors for any given 
target element that require higher and lower values, respectively. In the mountains example 
above, for instance, mountain ranges usually qualify most literally as irregular forms, whereas the 
more stylized interpretations involve inverted V-shapes, etc. In any case, it should be noted that 


any assigned target will have to be defendable on a visual basis. 


It is anticipated that when a truly significant geometric form is encountered in a 
target, such as in the Transamerica Pyramid, it will be accorded a higher triangle fuzzy value 
than would be accorded for mountains, because the Transamerica Pyramid is more 
unequivocally a triangular/pyramidal form. In this way, if a viewer registers a triangle ideogram 
for the Transamerica target, he will receive substantially more credit than if he uses a triangle 


ideogram for a mountain target. 


There is little doubt that the process of decomposing the concrete target elements into 
clusters of somewhat “imagined” viewer-derived abstract elements is going to be a 
noise—producing exercise. We’re assuming here, however, that (1) not every concrete feature 
can be decomposed into every abstract feature (e.g., the analyst will not typically be able to 
justify decomposing mountains into a cross—hatch (grid) geometry, but in all likelihood, he’ll feel 
comfortable in doing so for city), and (2) there is enough true viewer signal to effect significant 
overlap between targets and responses. Even at the abstract levels, one hopes the true signal line 
will rise above this very noisy background. The true advantage of this system, however, is that 
levels 1 or 2 or both can be turned off in the data base and the analyses rerun, if it appears that 


these initial hypotheses are incorrect. 


2. Abstract Descriptor Levels I: Levels 3 and 4 


Levels 3 and 4 are aimed primarily at viewers’ archetypal and visceral/qualitative 
impressions, respectively. While levels 3 and 4 are more information-rich than the primitive 
ideogramic geometries, they are still not sufficient in and of themselves to produce a coherent, 


concrete picture. Generally speaking, if an RV session is proceeding in an “on-line” fashion, a 
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geometric ideogram (levels 1 and 2) will be debriefed by descriptors from the level 3 and 4 


menus. 


Because levels 3 and 4, like levels 1 and 2, are generally response-derived rather 


than target-derived, the same rules for application pertain: 


(1) Targets should be as liberally but defendably interpreted (albeit with 
probably small fuzzy values) across as many descriptors as seem 
reasonable, in order to include all the viewers’ potential drawings and 
debriefs, while 


(2) Response elements should simply be chronicled in the descriptor list as 
they are literally observed. 


Again, the principle foreseeable problem will occur in trying to encode the target 
materials. The analyst will need to show some flexibility in understanding that a lush abundance 
of vegetation, for example, might be viscerally perceived by the viewer as the texture Suzzy, or the 
hard edge of a modern skyscraper might be experienced initially as smooth and Shiny/reflective. 
Perhaps the guiding principle here is for the analyst to be imaginative, but not excessive, in 
determining which descriptors merit fuzzy values. Any assigned target yw will have to be 
defendable on a visual basis. Again, as in the case with levels 1 and 2, some descriptors are 


more stylized than others and should probably be assigned lower values accordingly. 


It might be noticed that levels 3 and 4 contain some pairings of descriptors that are 
Opposite in meaning——e.g., rise versus flat, ordered versus disordered, complex versus simple, 
etc. This will enable the analyst to make assignments along opposing dimensions when they are 
present in targets or responses. For example, a mesa in the middle of an otherwise flat plain 
would presumably merit high values for both the rising and flat dimensions. If you recall, we had 
considered at one point having a single descriptor, elevation, for which a “0” would have 
indicated total flat or horizontality, and a “1” would have indicated total rise or verticality. In 
the mesa target example just cited, elevation would probably merit a 0.5, to indicate 50% flat and 
50% vertical rise; a 0.5 value along this single dimension would make this target indistinguishable 
from a target containing a single 45° slope mountain peak with no background except sky. This 
would appear to be conceptually incorrect because of the resultant loss of information across 
both dimensions in the case of the mesa. This kind of problem is the primary rationale for the 
inclusion of opposites. Clearly, it was not appropriate to supply an opposite for every descriptor, 
because most negatives are usually not asserted by viewers——e.g., if there is no perception of 


flowing, the viewer isn’t apt to say not flowing. 
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3. Concrete Descriptor Levels II: Levels 5, 6, and 7 


At level 5 and above, the descriptors are primarily target-derived rather than 
response—derived. Levels 5, 6, and 7 contain the bulk of the kinds of descriptors that we were 
accustomed to using in the previous twenty-bit list. Using the concrete descriptor levels poses a 
different set of problems for the analyst, because of the shift in the interpretive burden from 
target encoding to response encoding. The primary guidelines, therefore, for making assignments 


at all concrete levels are as follows: 


(1) Target elements should simply be chronicled in the descriptor list as they 
are literally observed, while 


(2) Response elements should be liberally but judiciously interpreted, with 
the actual numerical assignments made by taking into account, to 
whatever extent possible, the given viewer’s response biases and 
idiosyncrasies. 


This second point is a difficult one and requires further clarification. In an optimal 
situation, we would have complete and systematic retrospective knowledge-——a track record, as it 
were——for each viewer, in which we knew from post hoc analyses the probability for any given 
abstract response ideogram correctly matching any given concrete descriptor. Unfortunately this 
is quite a labor intensive investigation, and we haven’t had the opportunity to perform it as yet. 
Clearly this is a critical area of research to move into, if we are ever to have any ultimate success 
at objectifying the analytical process. 


What we do have at present are experienced analysts, who have looked at many 
years’ worth of data that have been obtained from the viewing population in question. Although 
we haven’t systematically codified our observations, we have some sense that when certain 
viewers say or draw certain abstract things, they are more apt to mean one thing as opposed to 
another. In general, the analyst should be fairly liberal and inclusive in his choices of which 
concrete descriptors are candidates for fuzzy values; where the awareness of response 
idiosyncrasies should come into play is in the assignment of fuzzy values for these choices. For 
example, if viewers 105 and 177 used the quality descriptor fuzzy in their responses, it would 
probably be reasonable in both cases to select vegetation as a possible interpretive descriptor for 
this impression. An experienced analyst, however, might be justified in determining that this 
interpretation is more valid for 177 than it is for 105, and would accord a higher numerical value 
for vegetation in the case of the former. Of course, if an experienced analyst has very high 
confidence that a certain combination of abstract elements simply cannot be interpreted as a 
certain candidate concrete descriptor for a given viewer, then that concrete descriptor should be 


eliminated from the candidate list altogether. Until the response biases of each viewer are 
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somehow systematized and objectified, it is certainly in the analyst’s best interest to have a sound 


grasp of the response styles of his viewing population insofar as it is possible. 


4. Concrete Descriptor Levels I: Levels 8, 9, and 10 


It is a matter for discussion as to whether levels 8, 9, and 10 are appropriate for 
inclusion with our core group of novice viewers at all, given that we infrequently see these levels 
of response detail. At present, therefore, they have been labeled as secondary rather than 
primary levels. Another matter for discussion is whether these sub-elements, as they are 
currently configured, belong at higher informational levels than 6 and 7. In some sense, in order 
to “build” the perception of a level 7 port, a viewer must have either consciously or 
unconsciously registered the impression that there are boats and perhaps piers (currently level 9) 
in the target. On the other hand, if a viewer says mosque, and is correct, this is much more 
useful from an information perspective than if he says religious. In any case, the current 


configuration of levels appears to require some form of readjustment. 


D. Overall Guidelines For Making the Numerical Assignments 


For targets, a numerical assignment pp is made for each descriptor in response to the 


following question: 
How visually important is this descriptor to this target? 


It should be noted that the descriptor set has been devised to define targets in terms of 
visual importance. If the analysis were to be re-framed along a different dimension of 


interest——e.g., conceptual, functional, allegorical, etc.--the descriptor set would have to be 
revised to address the new mission. 


For responses, 4 assignments for each descriptor are made in response to a different 


question: 


To what degree am I convinced that this descriptor represents this response 
element? 


Responses are coded according to this different criterion, because RV response composites 


are often not detailed enough to determine relative visual importance for any given element. 


The ’s themselves will typically consist of one-digit fuzzy numbers between 0 and 1 
(e.g., .1, .2, .3, etc.). In some cases, it may be appropriate to make two-digit assignments (e.g., 


-15, .25, .35, etc.); any finer assignments, however, would probably be meaningless. 
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The sequence for using the descriptor hierarchy is different for targets and responses. For 
targets, it is easiest to start with coding the concrete elements at the top of the hierarchy and 
work down to the abstract interpretations. For responses, the following sequence appears to be 


the most efficacious: 


(1) Go through the response and assign a “1” to every element explicitly stated by 
the viewer. 


(2) Go through the response again and assign j’s for elements that appear to be 
depicted rather unequivocally in the response (like a rectangle, for instance), 
but which is not labeled rectangular form or structure as such. These kinds of 
cases don’t merit “1s” per se (because they require some inference on the part 
of the analyst), but the ’s would be pretty high. Other situations that would 
probably fall into this category are instances when a viewer explicitly names 
something like Aill but equivocates by adding qualifiers like could be or might 
be. 


(3) Once you are comfortable with (1) your explicit “1” assignments throughout 
the levels and (2) the p assignments you have made at the lower levels 
(primarily 1-4), you should now be ready to make your “interpretive leaps” to 
the concrete descriptors in level 5 and beyond. Confidence will probably be 
fairly low in synthesizing these higher order composites, and the y’s should 
reflect this accordingly. Of course, if the weight of all your evidence really 
leads you to believe that the viewer has constructed a certain kind of 
scene-—~i.e., if your confidence is high—-then give it a high 4. The important 
idea here is that the judicious interpretive leap should be made whenever 
possible. The inclusion of the abstract levels does not relieve us of the 
responsibility of undertaking reasonable interpretation--rather, they are 
included at least in part to serve as heuristic devices for synthesizing higher 
order composites. There may be a few cases in which the response data are 
simply so sparse that they cannot lead to reasonable concrete interpretation so 


do not interpret, as long as it is defendable. These cases are anticipated to be 
exceptions. 


After the coding process has been finished for any given target or response, it is a useful 
exercise to take a Gestalt-oriented look at the coding. The important question to ask is 
following: does the descriptor portrait you have painted of this response or target strike you as 


reasonable? If not, try to evaluate where the portrait goes awry, and adjust the ’s accordingly. 


E. Specific Guidelines For Descriptor Use 


In the course of pilot attempts to use the descriptor list, certain preliminary guidelines have 
emerged for the use of specific descriptors. These are summarized by level and by descriptor 
below. 
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1. Level 1: 1-D Geometry 


a. Parallel Lines 


This descriptor should include curved as well as straight parallel lines. If an 
assignment is made for parallel lines, an attendant descriptive assignment for arc, horizontal 


lines, or vertical lines is also in order: 


b. Vertical Lines, Horizontal Lines, and Diagonal Lines 


For targets, these configurations of lines need to be interpreted as they appear 
in the 3-D projection of the photograph. Vertical lines are to be interpreted as perpendicular to 
the plane of the earth; horizontal lines are lines that are parallel to the plane of the earth; 


diagonal lines are lines that are projecting diagonally out of the plane of the earth. 


One might justifiably argue that in certain instances, this interpretation for 
targets (particularly with respect to diagonal lines) will result in the loss of some information. For 
example, if the target is a picture of an agricultural field that contains a vivid diagonal dividing 
line between two varieties of crops, we will not be allowed to make an assignment for diagonal 
lines because the line is in the plane of the earth. On first examination, this. would appear to bea 
potentially catastrophic loss of information. We must remember, however, that we have the 
visual correspondence analysis that will enable us to recapture much of this kind of photographic 
perspective information. In some sense, this division of the interpretive burden may be ideal, 
because we will be able to account for the true linear geometric information contained in the 


target site itself, as well as account for the given target photograph’s perspective view of that site. 


For responses, the intent of the viewer is the central consideration for the 


analyst in making an interpretation. For example, the following situation might occur: 


The viewer draws And Labels it 
this: this: 


7 “ Flat” 


In this case, the analyst would make no assignment for diagonal lines, because the viewer’s intent 


was to draw the edge of a plane in perspective. 
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c. Wave Form 


The wave form descriptor is intended to encompass both actual and stylized 
manifestations of wave forms. It will most usually receive a value for a viewer’s primitive and 


stylized depiction of water: 


)DKKA 


It should also receive a numerical value for any target manifestation of waves 


or ripples--i.e., the boat wake in the Gatun Lake target (1024) is a case in point. 


It is probably inappropriate to make wave form assignments based on a strict 
physics interpretation. For example, it would most likely not be a viewer’s intent to have the 


following possible drawings interpreted as wave forms: 


UW «© PLE 


2. Level 2: 2-D and 3-D Geometries 


a. Irregular Forms (Irregular Features) 


Because irregular forms can be visualized in almost every target, it is the 


current consensus that this descriptor should be used parsimoniously. 


3. Level 3: Archetypes 


a. Building(s) (Structure(s)) 


The building descriptor for targets should receive numerical assignments 
according to the prominence (i.e., visual presence or absence) of the buildings for the given 
target. In the Havana target (1005), for example, the buildings are quite prominent; in the 
Florence, Italy target (1056), on the other hand, one knows in the aggregate that the target is a 
city, but the buildings, themselves, are not individually prominent or resolvable. The building 
descriptor, therefore, while given a numerical value in both cases, would probably be higher in 


the former than in the latter. For responses, the following situation might occur: 
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The viewer draws And Labels it 
i this: 


ifn 
ql 


In this case, the analyst would be correct in making a numerical assignment for building(s) 


through inference. 


For targets, an assignment for the structure descriptor should occur only when 
a justification can be made for the misinterpretation of a natural feature as manmade feature. In 
a target photograph of a mesa, for example, the structure descriptor would merit some value, 
because of the knowledge that mesa geometries are often misinterpreted by viewers as building 
profiles. Response codings are usually less ambiguous, because a viewer will typically respond 


overtly with “structure,” in which case the numerical value would be “1.” 


b. Boundaries 


The boundaries descriptor is to be used in the sense of a distinct or discrete 
dividing line between different elements in a target or response. It may serve to demarcate one 
kind of land form from another, land from water, etc. It can also serve in the more traditional 


sense of a fence, a wall, etc. 


4. Level 4: Qualities 


a. Weathered 


The weathered descriptor is one that should be used parsimoniously; most of 
nature (and the targets used) should be regarded typically as falling somewhere in the range 
between 0 and 0.1. The intent of this descriptor is to address the strikingly weathered quality of 


such targets as the Grand Canyon or any number of ruins sites. 


b. Implied Texture 


All of the Implied Texture descriptors are to be used according to what the 


target element looks like, as opposed to what the analyst imagines the target element would feel 
like, 
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c. Fuzzy 


Under the heading of Other Visual, the fuzzy descriptor (a synonym for 
obscured) is intended to denote atmospheric obscurity. Under the heading of Implied Texture, 
however, the fuzzy descriptor is intended to address textures only. Obviously this is a descriptor 
that must be carefully examined in context. If a viewer writes “fuzzy” and clearly associates it 
with a response object, then the analyst should be inclined to assign a value to the Implied 
Texture fuzzy descriptor; if, however, a viewer writes “fuzzy” and clearly associates it with a 
horizon line or the word “sky,” then the analyst would be apt to assign a value for the obscured 
descriptor. 


d. Striated 


The striated descriptor is intended to suggest the presence of grooves, furrows, 
channels or narrow bands or lines. 


e. Hot and Humid 


Under the Jmplied Temperature heading, the two most potentially 
problematical descriptors are hot and humid. Both descriptors should only be used for targets if 
the visual indicators are overwhelming. An analyst might be justified in using hot, for example, if 
the target is a photograph of a desert scene with visible heat waves radiating off of the dunes. 
The humid descriptor might be used appropriately if the target were a photograph of a dense rain 
forest with water clearly dripping off of dense foliage. Clearly, the general rule is to be 
parsimonious in making assignments for these descriptors. (See the discussion for weathered, 
above.) 


5. Level 5 


a. [Plain] (Delta) 


This descriptor is to be used only in the literal sense as stated; for plains or 


deltas. It is not the intent that it should be used in the more general sense of flat land. 


b. Coastline 


This descriptor is to be used only in the sense of a typical ocean coastline, not 
as the coastline of a lake, or other inland body of water. 
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6. Level 7 


a. Wilderness 


The wilderness descriptor is perhaps most aptly thought of in the more general 


sense of untouched by the hand of man. 


F. Potential Problems 


One very persistent conceptual problem appears to be the one-to-many aspect of certain 
descriptors in the list, a situation which the list has no clear way of handling. For example, the 
descriptor repeat motif is one that could be applied to more than one element in a target or 
response. In a hypothetical target for instance, one would like to be able to say that there is a 
repeat motif of structures as a city, a repeat motif of irregular forms, inverted/or V-shapes, as 
mountains. Given the out-of-third-normal-form nature (to use data base parlance) of the 
current list, we are only able to make a single overall assessment for repeat motif; information 
seems to be lost (or distorted) in the breaking of the link, therefore, between repeat motif and its 
intended element. The same argument would probably be germane for the descriptors 


manmade and natural. 
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FIGURE 2-B CLUSTERS 4, 5, AND 11 
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FIGURE 4-B CLUSTERS 6 AND 8 
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FIGURE 5-B CLUSTERS 2, 7, AND 12 
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FIGURE 6-B CLUSTERS 13-16 
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FIGURE 7-B CLUSTER 17 
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FIGURE 8-B CLUSTER OF CLUSTERS 1-19 
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